File Systems — Storage Systems and Data Management Guide 1.0 documentation
File Systems — Storage Systems and Data Management Guide 1.0 documentation
File Systems — Storage Systems and Data Management Guide 1.0 documentation
File Systems
Understanding File Systems
A file system is a method used by operating systems to store, organize, and manage files on storage devices. It defines how data is stored, accessed, and organized on disks.
What is a File System?
A file system provides:
File Organization: Hierarchical structure for organizing files and directories
Metadata Management: Information about files (size, permissions, timestamps)
Space Management: Allocation and deallocation of disk space
Access Control: Security and permission mechanisms
Data Integrity: Protection against data corruption
File System Components
Core Components
Superblock: Contains metadata about the filesystem
Inode Table: Stores file metadata and pointers to data blocks
Data Blocks: Actual file content storage
Directory Structure: Hierarchical organization of files and folders
Journal: Transaction log for filesystem changes (in journaling filesystems)
File System Structure
Filesystem Layout
├── Superblock (filesystem metadata)
├── Group Descriptors (block group information)
├── Block Bitmap (free/used block tracking)
├── Inode Bitmap (free/used inode tracking)
├── Inode Table (file metadata)
└── Data Blocks (actual file content)
Common File Systems in Ubuntu 22.04
ext4 (Fourth Extended Filesystem)
The default filesystem for Ubuntu 22.04:
Features:
* Journaling for data integrity
* Large file and filesystem support (up to 1 EB)
* Backward compatibility with ext2/ext3
* Online defragmentation
* Delayed allocation
Use Cases:
* General purpose computing
* Desktop and server installations
* Boot partitions
* Home directories
# Create ext4 filesystem
sudo mkfs.ext4 /dev/sdb1
# Check ext4 filesystem
sudo fsck.ext4 /dev/sdb1
# Get ext4 filesystem information
sudo tune2fs -l /dev/sdb1
# Optimize ext4 filesystem
sudo tune2fs -o journal_data_writeback /dev/sdb1
XFS (eXtended File System)
High-performance 64-bit journaling filesystem:
Features:
* Excellent scalability
* Online resizing (grow only)
* Advanced quota management
* Allocation groups for parallel I/O
* Metadata journaling
Use Cases:
* Large files and databases
* High-performance computing
* Video editing and media storage
* Enterprise storage systems
# Create XFS filesystem
sudo mkfs.xfs /dev/sdb1
# Check XFS filesystem
sudo xfs_check /dev/sdb1
# Repair XFS filesystem
sudo xfs_repair /dev/sdb1
# Get XFS information
sudo xfs_info /dev/sdb1
# Resize XFS filesystem (grow only)
sudo xfs_growfs /mnt/xfs
Btrfs (B-tree File System)
Modern copy-on-write filesystem:
Features:
* Snapshots and cloning
* Built-in RAID support
* Compression (zlib, lzo, zstd)
* Checksumming for data integrity
* Online resizing (grow and shrink)
Use Cases:
* System snapshots
* Development environments
* Data deduplication scenarios
* Advanced storage management
# Create Btrfs filesystem
sudo mkfs.btrfs /dev/sdb1
# Mount with compression
sudo mount -o compress=zstd /dev/sdb1 /mnt/btrfs
# Create snapshot
sudo btrfs subvolume snapshot /mnt/btrfs /mnt/btrfs/snapshot
# List subvolumes
sudo btrfs subvolume list /mnt/btrfs
ZFS (Zettabyte File System)
Advanced filesystem with built-in volume management:
Features:
* Built-in RAID (RAID-Z)
* Snapshots and clones
* Data deduplication
* Compression
* End-to-end checksumming
Installation and Use:
# Install ZFS on Ubuntu 22.04
sudo apt update
sudo apt install zfsutils-linux
# Create ZFS pool
sudo zpool create mypool /dev/sdb
# Create ZFS dataset
sudo zfs create mypool/data
# Enable compression
sudo zfs set compression=lz4 mypool/data
# Create snapshot
sudo zfs snapshot mypool/data@snapshot1
FAT32 and NTFS
FAT32: Universal compatibility, limited file size (4GB max)
NTFS: Windows filesystem with advanced features
# Create FAT32 filesystem
sudo mkfs.fat -F32 /dev/sdb1
# Create NTFS filesystem
sudo mkfs.ntfs /dev/sdb1
# Mount NTFS with full permissions
sudo mount -t ntfs-3g /dev/sdb1 /mnt/ntfs -o permissions
File System Operations in Ubuntu 22.04
Creating File Systems
# ext4 with custom options
sudo mkfs.ext4 -L "MyData" -m 1 /dev/sdb1
# XFS with custom block size
sudo mkfs.xfs -f -b size=4096 /dev/sdb1
# Btrfs with multiple devices (RAID)
sudo mkfs.btrfs -d raid1 -m raid1 /dev/sdb1 /dev/sdc1
# Set filesystem label
sudo e2label /dev/sdb1 "DataDisk"
Mounting File Systems
# Basic mounting
sudo mount /dev/sdb1 /mnt/data
# Mount with specific options
sudo mount -o rw,noatime,discard /dev/sdb1 /mnt/data
# Mount by UUID (preferred method)
sudo mount UUID=12345678-1234-1234-1234-123456789abc /mnt/data
# Mount with user permissions
sudo mount -o uid=1000,gid=1000 /dev/sdb1 /mnt/data
Checking and Repairing File Systems
# Check filesystem (read-only)
sudo fsck -n /dev/sdb1
# Automatic repair
sudo fsck -p /dev/sdb1
# Force check and repair
sudo fsck -f /dev/sdb1
# Check specific filesystem types
sudo fsck.ext4 /dev/sdb1
sudo xfs_check /dev/sdb1
sudo btrfs check /dev/sdb1
File System Monitoring
Monitoring Tools and Commands
# Real-time filesystem usage
watch df -h
# Inode usage
df -i
# Detailed filesystem information
stat -f /
# Monitor filesystem I/O
sudo iotop -o
# Check filesystem fragmentation (ext4)
sudo e4defrag -c /dev/sdb1
# Btrfs filesystem usage
sudo btrfs filesystem usage /mnt/btrfs
Frequently Asked Questions
Q: Which filesystem should I choose for Ubuntu 22.04?
A: Filesystem selection depends on your use case:
ext4: Best general-purpose choice, default for Ubuntu
XFS: Large files, databases, high-performance needs
Btrfs: Need snapshots, compression, or advanced features
ZFS: Enterprise features, data integrity critical
# For most users (recommended)
sudo mkfs.ext4 /dev/sdb1
# For large files and databases
sudo mkfs.xfs /dev/sdb1
# For snapshots and modern features
sudo mkfs.btrfs /dev/sdb1
Q: How do I convert between filesystems?
A: Filesystem conversion requires backup and restore:
# 1. Backup data
sudo rsync -av /mnt/source/ /backup/
# 2. Unmount and recreate filesystem
sudo umount /mnt/source
sudo mkfs.ext4 /dev/sdb1
# 3. Mount and restore data
sudo mount /dev/sdb1 /mnt/source
sudo rsync -av /backup/ /mnt/source/
Q: How do I optimize filesystem performance?
A: Several optimization techniques:
# For SSDs - enable TRIM
sudo mount -o discard /dev/sdb1 /mnt/data
# Disable access time updates
sudo mount -o noatime /dev/sdb1 /mnt/data
# For databases - use direct I/O
sudo mount -o barrier=0 /dev/sdb1 /mnt/database
# Optimize ext4 for SSDs
sudo tune2fs -o discard /dev/sdb1
Q: How do I recover deleted files?
A: Recovery methods depend on the filesystem:
# Install recovery tools
sudo apt install testdisk photorec extundelete
# For ext4 filesystems
sudo extundelete /dev/sdb1 --restore-all
# General purpose recovery
sudo photorec /dev/sdb1
# For immediate action after deletion
sudo grep -a -B25 -A25 'text from deleted file' /dev/sdb1
Coding Examples
Python Filesystem Analyzer
#!/usr/bin/env python3
"""
Filesystem analyzer for Ubuntu 22.04
"""
import os
import subprocess
import json
import time
from pathlib import Path
class FilesystemAnalyzer:
def __init__(self):
self.mountpoints = self.get_mountpoints()
def get_mountpoints(self):
"""Get all mounted filesystems"""
mountpoints = []
try:
with open('/proc/mounts', 'r') as f:
for line in f:
parts = line.strip().split()
if len(parts) >= 3 and parts[0].startswith('/dev/'):
mountpoints.append({
'device': parts[0],
'mountpoint': parts[1],
'filesystem': parts[2],
'options': parts[3]
})
except Exception as e:
print(f"Error reading mount points: {e}")
return mountpoints
def analyze_filesystem(self, path='/'):
"""Analyze filesystem usage and characteristics"""
try:
# Get basic filesystem statistics
statvfs = os.statvfs(path)
total_size = statvfs.f_frsize * statvfs.f_blocks
free_size = statvfs.f_frsize * statvfs.f_bavail
used_size = total_size - free_size
# Get inode information
total_inodes = statvfs.f_files
free_inodes = statvfs.f_favail
used_inodes = total_inodes - free_inodes
return {
'path': path,
'total_size_gb': round(total_size / (1024**3), 2),
'used_size_gb': round(used_size / (1024**3), 2),
'free_size_gb': round(free_size / (1024**3), 2),
'usage_percent': round((used_size / total_size) * 100, 2),
'total_inodes': total_inodes,
'used_inodes': used_inodes,
'free_inodes': free_inodes,
'inode_usage_percent': round((used_inodes / total_inodes) * 100, 2) if total_inodes > 0 else 0
}
except Exception as e:
return {'error': str(e)}
def get_largest_files(self, path='/', min_size_mb=100, count=10):
"""Find largest files in a directory tree"""
large_files = []
min_size_bytes = min_size_mb * 1024 * 1024
try:
for root, dirs, files in os.walk(path):
# Skip certain directories to avoid permission errors
dirs[:] = [d for d in dirs if not d.startswith('.') and d not in ['proc', 'sys', 'dev']]
for file in files:
try:
file_path = os.path.join(root, file)
file_size = os.path.getsize(file_path)
if file_size >= min_size_bytes:
large_files.append({
'path': file_path,
'size_mb': round(file_size / (1024**2), 2),
'size_gb': round(file_size / (1024**3), 2)
})
except (OSError, IOError):
continue # Skip files we can't access
# Sort by size and return top files
large_files.sort(key=lambda x: x['size_mb'], reverse=True)
return large_files[:count]
except Exception as e:
return [{'error': str(e)}]
def get_directory_sizes(self, path='/', max_depth=2):
"""Get sizes of directories"""
directory_sizes = []
try:
result = subprocess.run(['du', '-h', f'--max-depth={max_depth}', path],
capture_output=True, text=True)
for line in result.stdout.strip().split('\n'):
if line:
parts = line.split('\t')
if len(parts) == 2:
size, dir_path = parts
directory_sizes.append({
'path': dir_path,
'size': size
})
return directory_sizes
except Exception as e:
return [{'error': str(e)}]
def check_filesystem_health(self, device):
"""Check filesystem health using fsck"""
try:
# Run read-only check
result = subprocess.run(['fsck', '-n', device],
capture_output=True, text=True)
return {
'device': device,
'status': 'clean' if result.returncode == 0 else 'errors_found',
'output': result.stdout,
'errors': result.stderr
}
except Exception as e:
return {'device': device, 'error': str(e)}
def monitor_filesystem_performance(self, duration=10):
"""Monitor filesystem I/O performance"""
try:
# Start iostat monitoring
process = subprocess.Popen(['iostat', '-x', '1', str(duration)],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True)
output, error = process.communicate()
return {
'duration': duration,
'iostat_output': output,
'error': error if error else None
}
except Exception as e:
return {'error': str(e)}
def generate_report(self):
"""Generate comprehensive filesystem report"""
report = {
'timestamp': time.strftime('%Y-%m-%d %H:%M:%S'),
'hostname': os.uname().nodename,
'mountpoints': self.mountpoints,
'filesystem_analysis': [],
'large_files': [],
'directory_sizes': []
}
# Analyze each mounted filesystem
for mount in self.mountpoints:
analysis = self.analyze_filesystem(mount['mountpoint'])
analysis['device'] = mount['device']
analysis['filesystem_type'] = mount['filesystem']
report['filesystem_analysis'].append(analysis)
# Find large files in home and var directories
for path in ['/home', '/var']:
if os.path.exists(path):
large_files = self.get_largest_files(path, min_size_mb=50, count=5)
report['large_files'].extend(large_files)
# Get directory sizes for common directories
for path in ['/', '/home', '/var', '/usr']:
if os.path.exists(path):
dir_sizes = self.get_directory_sizes(path, max_depth=2)
report['directory_sizes'].extend(dir_sizes)
return report
# Example usage
if __name__ == "__main__":
analyzer = FilesystemAnalyzer()
# Generate and display report
report = analyzer.generate_report()
print(json.dumps(report, indent=2))
# Save report to file
with open(f'/tmp/filesystem_report_{int(time.time())}.json', 'w') as f:
json.dump(report, f, indent=2)
Bash Filesystem Management Script
#!/bin/bash
# filesystem_manager.sh - Comprehensive filesystem management for Ubuntu 22.04
# Configuration
BACKUP_DIR="/backup"
LOG_FILE="/var/log/filesystem_manager.log"
# Colors
RED='[0;31m'
GREEN='[0;32m'
YELLOW='[1;33m'
BLUE='[0;34m'
NC='[0m'
# Logging function
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$LOG_FILE"
}
# Check if running as root
check_root() {
if [[ $EUID -ne 0 ]]; then
echo -e "${RED}This script must be run as root${NC}"
exit 1
fi
}
# Display filesystem information
show_filesystem_info() {
echo -e "${BLUE}=== Filesystem Information ===${NC}"
echo "Mounted Filesystems:"
df -hT | grep -v tmpfs
echo ""
echo "Filesystem Types:"
lsblk -f
echo ""
echo "Mount Points:"
mount | grep "^/dev" | column -t
echo ""
}
# Create filesystem with optimal settings
create_filesystem() {
local device="$1"
local fstype="$2"
local label="$3"
if [ -z "$device" ] || [ -z "$fstype" ]; then
echo "Usage: create_filesystem <device> <filesystem_type> [label]"
echo "Supported types: ext4, xfs, btrfs"
return 1
fi
echo -e "${YELLOW}Creating $fstype filesystem on $device${NC}"
case "$fstype" in
"ext4")
if [ -n "$label" ]; then
mkfs.ext4 -L "$label" -m 1 "$device"
else
mkfs.ext4 -m 1 "$device"
fi
# Optimize for SSD if detected
if [[ $(cat /sys/block/$(basename $device | sed 's/[0-9]*$//')/queue/rotational) == "0" ]]; then
tune2fs -o discard "$device"
echo "SSD optimizations applied"
fi
;;
"xfs")
if [ -n "$label" ]; then
mkfs.xfs -f -L "$label" "$device"
else
mkfs.xfs -f "$device"
fi
;;
"btrfs")
if [ -n "$label" ]; then
mkfs.btrfs -f -L "$label" "$device"
else
mkfs.btrfs -f "$device"
fi
;;
*)
echo -e "${RED}Unsupported filesystem type: $fstype${NC}"
return 1
;;
esac
log_message "Created $fstype filesystem on $device"
echo -e "${GREEN}Filesystem created successfully${NC}"
}
# Mount filesystem with optimal options
mount_filesystem() {
local device="$1"
local mountpoint="$2"
local fstype="$3"
if [ -z "$device" ] || [ -z "$mountpoint" ]; then
echo "Usage: mount_filesystem <device> <mountpoint> [filesystem_type]"
return 1
fi
# Create mountpoint if it doesn't exist
mkdir -p "$mountpoint"
# Detect filesystem type if not provided
if [ -z "$fstype" ]; then
fstype=$(blkid -o value -s TYPE "$device")
fi
# Set optimal mount options based on filesystem type
case "$fstype" in
"ext4")
mount_options="defaults,noatime"
# Add discard for SSDs
if [[ $(cat /sys/block/$(basename $device | sed 's/[0-9]*$//')/queue/rotational) == "0" ]]; then
mount_options="$mount_options,discard"
fi
;;
"xfs")
mount_options="defaults,noatime"
;;
"btrfs")
mount_options="defaults,noatime,compress=zstd"
;;
*)
mount_options="defaults"
;;
esac
echo -e "${YELLOW}Mounting $device at $mountpoint with options: $mount_options${NC}"
if mount -o "$mount_options" "$device" "$mountpoint"; then
log_message "Mounted $device at $mountpoint"
echo -e "${GREEN}Filesystem mounted successfully${NC}"
# Add to fstab for permanent mounting
uuid=$(blkid -o value -s UUID "$device")
if [ -n "$uuid" ]; then
echo "UUID=$uuid $mountpoint $fstype $mount_options 0 2" >> /etc/fstab
echo "Added to /etc/fstab for automatic mounting"
fi
else
echo -e "${RED}Failed to mount filesystem${NC}"
return 1
fi
}
# Check filesystem health
check_filesystem_health() {
local device="$1"
if [ -z "$device" ]; then
echo "Usage: check_filesystem_health <device>"
return 1
fi
echo -e "${BLUE}=== Checking filesystem health for $device ===${NC}"
# Get filesystem type
fstype=$(blkid -o value -s TYPE "$device")
case "$fstype" in
"ext4"|"ext3"|"ext2")
echo "Running fsck for ext filesystem..."
fsck.ext4 -n "$device"
;;
"xfs")
echo "Running xfs_check..."
xfs_check "$device"
;;
"btrfs")
echo "Running btrfs check..."
btrfs check "$device"
;;
*)
echo "Running generic fsck..."
fsck -n "$device"
;;
esac
log_message "Filesystem health check completed for $device"
}
# Optimize filesystem performance
optimize_filesystem() {
local mountpoint="$1"
if [ -z "$mountpoint" ]; then
echo "Usage: optimize_filesystem <mountpoint>"
return 1
fi
echo -e "${BLUE}=== Optimizing filesystem at $mountpoint ===${NC}"
# Get device and filesystem type
device=$(df "$mountpoint" | tail -1 | awk '{print $1}')
fstype=$(df -T "$mountpoint" | tail -1 | awk '{print $2}')
case "$fstype" in
"ext4")
echo "Optimizing ext4 filesystem..."
# Defragment if needed
e4defrag -c "$mountpoint"
# Optimize inode allocation
tune2fs -o journal_data_writeback "$device"
# Enable TRIM for SSDs
if [[ $(cat /sys/block/$(basename $device | sed 's/[0-9]*$//')/queue/rotational) == "0" ]]; then
tune2fs -o discard "$device"
echo "SSD optimizations applied"
fi
;;
"xfs")
echo "Optimizing XFS filesystem..."
# XFS is generally well-optimized by default
# Check if online defragmentation is needed
xfs_fsr -v "$mountpoint"
;;
"btrfs")
echo "Optimizing Btrfs filesystem..."
# Balance filesystem
btrfs balance start "$mountpoint"
# Defragment
btrfs filesystem defragment -r "$mountpoint"
;;
esac
log_message "Filesystem optimization completed for $mountpoint"
echo -e "${GREEN}Optimization completed${NC}"
}
# Backup filesystem
backup_filesystem() {
local source="$1"
local backup_name="$2"
if [ -z "$source" ]; then
echo "Usage: backup_filesystem <source_mountpoint> [backup_name]"
return 1
fi
if [ -z "$backup_name" ]; then
backup_name="filesystem_backup_$(date +%Y%m%d_%H%M%S)"
fi
backup_path="$BACKUP_DIR/$backup_name"
echo -e "${YELLOW}Creating backup of $source to $backup_path${NC}"
# Create backup directory
mkdir -p "$backup_path"
# Use rsync for efficient backup
if rsync -avH --progress "$source/" "$backup_path/"; then
log_message "Backup created: $source -> $backup_path"
echo -e "${GREEN}Backup completed successfully${NC}"
# Create metadata file
cat > "$backup_path/.backup_info" << EOF
Source: $source
Backup Date: $(date)
Backup Size: $(du -sh "$backup_path" | cut -f1)
EOF
else
echo -e "${RED}Backup failed${NC}"
return 1
fi
}
# Monitor filesystem usage
monitor_filesystem() {
echo -e "${BLUE}=== Filesystem Usage Monitor ===${NC}"
while true; do
clear
echo "Filesystem Usage (updated every 5 seconds) - Press Ctrl+C to exit"
echo "================================================================"
df -h | grep -E "^Filesystem|^/dev"
echo ""
echo "I/O Statistics:"
iostat -x 1 1 | tail -n +4
echo ""
echo "Top 5 largest directories in /:"
du -h --max-depth=1 / 2>/dev/null | sort -hr | head -5
sleep 5
done
}
# Main menu
case "${1:-help}" in
"info")
show_filesystem_info
;;
"create")
check_root
create_filesystem "$2" "$3" "$4"
;;
"mount")
check_root
mount_filesystem "$2" "$3" "$4"
;;
"check")
check_root
check_filesystem_health "$2"
;;
"optimize")
check_root
optimize_filesystem "$2"
;;
"backup")
check_root
backup_filesystem "$2" "$3"
;;
"monitor")
monitor_filesystem
;;
"help"|*)
echo "Filesystem Manager for Ubuntu 22.04"
echo "Usage: $0 [command] [options]"
echo ""
echo "Commands:"
echo " info - Show filesystem information"
echo " create <device> <type> [label] - Create filesystem"
echo " mount <device> <mountpoint> - Mount filesystem with optimal options"
echo " check <device> - Check filesystem health"
echo " optimize <mountpoint> - Optimize filesystem performance"
echo " backup <source> [name] - Backup filesystem"
echo " monitor - Monitor filesystem usage"
echo ""
echo "Supported filesystem types: ext4, xfs, btrfs"
echo ""
echo "Examples:"
echo " $0 create /dev/sdb1 ext4 MyData"
echo " $0 mount /dev/sdb1 /mnt/data"
echo " $0 check /dev/sdb1"
echo " $0 optimize /home"
echo " $0 backup /home home_backup"
;;
esac
Best Practices for File Systems
Choosing the Right Filesystem
Decision Matrix:
Use Case | Recommended Filesystem
----------------------------|----------------------
General desktop/server | ext4
Large files/databases | XFS
Snapshots/modern features | Btrfs
Maximum data integrity | ZFS
Cross-platform compatibility| FAT32/NTFS
High-performance computing | XFS or ext4
Performance Optimization
Mount Options:
# For general use
mount -o defaults,noatime /dev/sdb1 /mnt/data
# For SSDs
mount -o defaults,noatime,discard /dev/sdb1 /mnt/data
# For databases
mount -o defaults,noatime,barrier=0 /dev/sdb1 /mnt/database
I/O Scheduler:
# For SSDs
echo none > /sys/block/sda/queue/scheduler
# For HDDs
echo mq-deadline > /sys/block/sda/queue/scheduler
Filesystem Tuning:
# Optimize ext4 for SSDs
tune2fs -o discard /dev/sda1
# Reduce reserved space (ext4)
tune2fs -m 1 /dev/sda1
Security and Maintenance
Regular Checks:
# Schedule regular filesystem checks
echo "0 3 * * 0 /sbin/fsck -A -f" >> /etc/crontab
Backup Strategy:
# Automated backups
rsync -avH /home/ /backup/home/
# Filesystem snapshots (Btrfs)
btrfs subvolume snapshot /home /home/.snapshots/$(date +%Y%m%d)
Monitoring:
# Set up disk usage alerts
echo "df -h | awk 'NR>1 {if(\$5+0 > 85) print \$0}' | mail -s 'Disk Alert' admin@example.com" | crontab -
File Systems — Storage Systems and Data Management Guide 1.0 documentation
File Systems — Storage Systems and Data Management Guide 1.0 documentation
File Systems
Understanding File Systems
A file system is a method used by operating systems to store, organize, and manage files on storage devices. It defines how data is stored, accessed, and organized on disks.
What is a File System?
A file system provides:
File Organization: Hierarchical structure for organizing files and directories
Metadata Management: Information about files (size, permissions, timestamps)
Space Management: Allocation and deallocation of disk space
Access Control: Security and permission mechanisms
Data Integrity: Protection against data corruption
File System Components
Core Components
Superblock: Contains metadata about the filesystem
Inode Table: Stores file metadata and pointers to data blocks
Data Blocks: Actual file content storage
Directory Structure: Hierarchical organization of files and folders
Journal: Transaction log for filesystem changes (in journaling filesystems)
File System Structure
Filesystem Layout
├── Superblock (filesystem metadata)
├── Group Descriptors (block group information)
├── Block Bitmap (free/used block tracking)
├── Inode Bitmap (free/used inode tracking)
├── Inode Table (file metadata)
└── Data Blocks (actual file content)
Common File Systems in Ubuntu 22.04
ext4 (Fourth Extended Filesystem)
The default filesystem for Ubuntu 22.04:
Features:
* Journaling for data integrity
* Large file and filesystem support (up to 1 EB)
* Backward compatibility with ext2/ext3
* Online defragmentation
* Delayed allocation
Use Cases:
* General purpose computing
* Desktop and server installations
* Boot partitions
* Home directories
# Create ext4 filesystem
sudo mkfs.ext4 /dev/sdb1
# Check ext4 filesystem
sudo fsck.ext4 /dev/sdb1
# Get ext4 filesystem information
sudo tune2fs -l /dev/sdb1
# Optimize ext4 filesystem
sudo tune2fs -o journal_data_writeback /dev/sdb1
XFS (eXtended File System)
High-performance 64-bit journaling filesystem:
Features:
* Excellent scalability
* Online resizing (grow only)
* Advanced quota management
* Allocation groups for parallel I/O
* Metadata journaling
Use Cases:
* Large files and databases
* High-performance computing
* Video editing and media storage
* Enterprise storage systems
# Create XFS filesystem
sudo mkfs.xfs /dev/sdb1
# Check XFS filesystem
sudo xfs_check /dev/sdb1
# Repair XFS filesystem
sudo xfs_repair /dev/sdb1
# Get XFS information
sudo xfs_info /dev/sdb1
# Resize XFS filesystem (grow only)
sudo xfs_growfs /mnt/xfs
Btrfs (B-tree File System)
Modern copy-on-write filesystem:
Features:
* Snapshots and cloning
* Built-in RAID support
* Compression (zlib, lzo, zstd)
* Checksumming for data integrity
* Online resizing (grow and shrink)
Use Cases:
* System snapshots
* Development environments
* Data deduplication scenarios
* Advanced storage management
# Create Btrfs filesystem
sudo mkfs.btrfs /dev/sdb1
# Mount with compression
sudo mount -o compress=zstd /dev/sdb1 /mnt/btrfs
# Create snapshot
sudo btrfs subvolume snapshot /mnt/btrfs /mnt/btrfs/snapshot
# List subvolumes
sudo btrfs subvolume list /mnt/btrfs
ZFS (Zettabyte File System)
Advanced filesystem with built-in volume management:
Features:
* Built-in RAID (RAID-Z)
* Snapshots and clones
* Data deduplication
* Compression
* End-to-end checksumming
Installation and Use:
# Install ZFS on Ubuntu 22.04
sudo apt update
sudo apt install zfsutils-linux
# Create ZFS pool
sudo zpool create mypool /dev/sdb
# Create ZFS dataset
sudo zfs create mypool/data
# Enable compression
sudo zfs set compression=lz4 mypool/data
# Create snapshot
sudo zfs snapshot mypool/data@snapshot1
FAT32 and NTFS
FAT32: Universal compatibility, limited file size (4GB max)
NTFS: Windows filesystem with advanced features
# Create FAT32 filesystem
sudo mkfs.fat -F32 /dev/sdb1
# Create NTFS filesystem
sudo mkfs.ntfs /dev/sdb1
# Mount NTFS with full permissions
sudo mount -t ntfs-3g /dev/sdb1 /mnt/ntfs -o permissions
File System Operations in Ubuntu 22.04
Creating File Systems
# ext4 with custom options
sudo mkfs.ext4 -L "MyData" -m 1 /dev/sdb1
# XFS with custom block size
sudo mkfs.xfs -f -b size=4096 /dev/sdb1
# Btrfs with multiple devices (RAID)
sudo mkfs.btrfs -d raid1 -m raid1 /dev/sdb1 /dev/sdc1
# Set filesystem label
sudo e2label /dev/sdb1 "DataDisk"
Mounting File Systems
# Basic mounting
sudo mount /dev/sdb1 /mnt/data
# Mount with specific options
sudo mount -o rw,noatime,discard /dev/sdb1 /mnt/data
# Mount by UUID (preferred method)
sudo mount UUID=12345678-1234-1234-1234-123456789abc /mnt/data
# Mount with user permissions
sudo mount -o uid=1000,gid=1000 /dev/sdb1 /mnt/data
Checking and Repairing File Systems
# Check filesystem (read-only)
sudo fsck -n /dev/sdb1
# Automatic repair
sudo fsck -p /dev/sdb1
# Force check and repair
sudo fsck -f /dev/sdb1
# Check specific filesystem types
sudo fsck.ext4 /dev/sdb1
sudo xfs_check /dev/sdb1
sudo btrfs check /dev/sdb1
File System Monitoring
Monitoring Tools and Commands
# Real-time filesystem usage
watch df -h
# Inode usage
df -i
# Detailed filesystem information
stat -f /
# Monitor filesystem I/O
sudo iotop -o
# Check filesystem fragmentation (ext4)
sudo e4defrag -c /dev/sdb1
# Btrfs filesystem usage
sudo btrfs filesystem usage /mnt/btrfs
Frequently Asked Questions
Q: Which filesystem should I choose for Ubuntu 22.04?
A: Filesystem selection depends on your use case:
ext4: Best general-purpose choice, default for Ubuntu
XFS: Large files, databases, high-performance needs
Btrfs: Need snapshots, compression, or advanced features
ZFS: Enterprise features, data integrity critical
# For most users (recommended)
sudo mkfs.ext4 /dev/sdb1
# For large files and databases
sudo mkfs.xfs /dev/sdb1
# For snapshots and modern features
sudo mkfs.btrfs /dev/sdb1
Q: How do I convert between filesystems?
A: Filesystem conversion requires backup and restore:
# 1. Backup data
sudo rsync -av /mnt/source/ /backup/
# 2. Unmount and recreate filesystem
sudo umount /mnt/source
sudo mkfs.ext4 /dev/sdb1
# 3. Mount and restore data
sudo mount /dev/sdb1 /mnt/source
sudo rsync -av /backup/ /mnt/source/
Q: How do I optimize filesystem performance?
A: Several optimization techniques:
# For SSDs - enable TRIM
sudo mount -o discard /dev/sdb1 /mnt/data
# Disable access time updates
sudo mount -o noatime /dev/sdb1 /mnt/data
# For databases - use direct I/O
sudo mount -o barrier=0 /dev/sdb1 /mnt/database
# Optimize ext4 for SSDs
sudo tune2fs -o discard /dev/sdb1
Q: How do I recover deleted files?
A: Recovery methods depend on the filesystem:
# Install recovery tools
sudo apt install testdisk photorec extundelete
# For ext4 filesystems
sudo extundelete /dev/sdb1 --restore-all
# General purpose recovery
sudo photorec /dev/sdb1
# For immediate action after deletion
sudo grep -a -B25 -A25 'text from deleted file' /dev/sdb1
Coding Examples
Python Filesystem Analyzer
#!/usr/bin/env python3
"""
Filesystem analyzer for Ubuntu 22.04
"""
import os
import subprocess
import json
import time
from pathlib import Path
class FilesystemAnalyzer:
def __init__(self):
self.mountpoints = self.get_mountpoints()
def get_mountpoints(self):
"""Get all mounted filesystems"""
mountpoints = []
try:
with open('/proc/mounts', 'r') as f:
for line in f:
parts = line.strip().split()
if len(parts) >= 3 and parts[0].startswith('/dev/'):
mountpoints.append({
'device': parts[0],
'mountpoint': parts[1],
'filesystem': parts[2],
'options': parts[3]
})
except Exception as e:
print(f"Error reading mount points: {e}")
return mountpoints
def analyze_filesystem(self, path='/'):
"""Analyze filesystem usage and characteristics"""
try:
# Get basic filesystem statistics
statvfs = os.statvfs(path)
total_size = statvfs.f_frsize * statvfs.f_blocks
free_size = statvfs.f_frsize * statvfs.f_bavail
used_size = total_size - free_size
# Get inode information
total_inodes = statvfs.f_files
free_inodes = statvfs.f_favail
used_inodes = total_inodes - free_inodes
return {
'path': path,
'total_size_gb': round(total_size / (1024**3), 2),
'used_size_gb': round(used_size / (1024**3), 2),
'free_size_gb': round(free_size / (1024**3), 2),
'usage_percent': round((used_size / total_size) * 100, 2),
'total_inodes': total_inodes,
'used_inodes': used_inodes,
'free_inodes': free_inodes,
'inode_usage_percent': round((used_inodes / total_inodes) * 100, 2) if total_inodes > 0 else 0
}
except Exception as e:
return {'error': str(e)}
def get_largest_files(self, path='/', min_size_mb=100, count=10):
"""Find largest files in a directory tree"""
large_files = []
min_size_bytes = min_size_mb * 1024 * 1024
try:
for root, dirs, files in os.walk(path):
# Skip certain directories to avoid permission errors
dirs[:] = [d for d in dirs if not d.startswith('.') and d not in ['proc', 'sys', 'dev']]
for file in files:
try:
file_path = os.path.join(root, file)
file_size = os.path.getsize(file_path)
if file_size >= min_size_bytes:
large_files.append({
'path': file_path,
'size_mb': round(file_size / (1024**2), 2),
'size_gb': round(file_size / (1024**3), 2)
})
except (OSError, IOError):
continue # Skip files we can't access
# Sort by size and return top files
large_files.sort(key=lambda x: x['size_mb'], reverse=True)
return large_files[:count]
except Exception as e:
return [{'error': str(e)}]
def get_directory_sizes(self, path='/', max_depth=2):
"""Get sizes of directories"""
directory_sizes = []
try:
result = subprocess.run(['du', '-h', f'--max-depth={max_depth}', path],
capture_output=True, text=True)
for line in result.stdout.strip().split('\n'):
if line:
parts = line.split('\t')
if len(parts) == 2:
size, dir_path = parts
directory_sizes.append({
'path': dir_path,
'size': size
})
return directory_sizes
except Exception as e:
return [{'error': str(e)}]
def check_filesystem_health(self, device):
"""Check filesystem health using fsck"""
try:
# Run read-only check
result = subprocess.run(['fsck', '-n', device],
capture_output=True, text=True)
return {
'device': device,
'status': 'clean' if result.returncode == 0 else 'errors_found',
'output': result.stdout,
'errors': result.stderr
}
except Exception as e:
return {'device': device, 'error': str(e)}
def monitor_filesystem_performance(self, duration=10):
"""Monitor filesystem I/O performance"""
try:
# Start iostat monitoring
process = subprocess.Popen(['iostat', '-x', '1', str(duration)],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True)
output, error = process.communicate()
return {
'duration': duration,
'iostat_output': output,
'error': error if error else None
}
except Exception as e:
return {'error': str(e)}
def generate_report(self):
"""Generate comprehensive filesystem report"""
report = {
'timestamp': time.strftime('%Y-%m-%d %H:%M:%S'),
'hostname': os.uname().nodename,
'mountpoints': self.mountpoints,
'filesystem_analysis': [],
'large_files': [],
'directory_sizes': []
}
# Analyze each mounted filesystem
for mount in self.mountpoints:
analysis = self.analyze_filesystem(mount['mountpoint'])
analysis['device'] = mount['device']
analysis['filesystem_type'] = mount['filesystem']
report['filesystem_analysis'].append(analysis)
# Find large files in home and var directories
for path in ['/home', '/var']:
if os.path.exists(path):
large_files = self.get_largest_files(path, min_size_mb=50, count=5)
report['large_files'].extend(large_files)
# Get directory sizes for common directories
for path in ['/', '/home', '/var', '/usr']:
if os.path.exists(path):
dir_sizes = self.get_directory_sizes(path, max_depth=2)
report['directory_sizes'].extend(dir_sizes)
return report
# Example usage
if __name__ == "__main__":
analyzer = FilesystemAnalyzer()
# Generate and display report
report = analyzer.generate_report()
print(json.dumps(report, indent=2))
# Save report to file
with open(f'/tmp/filesystem_report_{int(time.time())}.json', 'w') as f:
json.dump(report, f, indent=2)
Bash Filesystem Management Script
#!/bin/bash
# filesystem_manager.sh - Comprehensive filesystem management for Ubuntu 22.04
# Configuration
BACKUP_DIR="/backup"
LOG_FILE="/var/log/filesystem_manager.log"
# Colors
RED='[0;31m'
GREEN='[0;32m'
YELLOW='[1;33m'
BLUE='[0;34m'
NC='[0m'
# Logging function
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$LOG_FILE"
}
# Check if running as root
check_root() {
if [[ $EUID -ne 0 ]]; then
echo -e "${RED}This script must be run as root${NC}"
exit 1
fi
}
# Display filesystem information
show_filesystem_info() {
echo -e "${BLUE}=== Filesystem Information ===${NC}"
echo "Mounted Filesystems:"
df -hT | grep -v tmpfs
echo ""
echo "Filesystem Types:"
lsblk -f
echo ""
echo "Mount Points:"
mount | grep "^/dev" | column -t
echo ""
}
# Create filesystem with optimal settings
create_filesystem() {
local device="$1"
local fstype="$2"
local label="$3"
if [ -z "$device" ] || [ -z "$fstype" ]; then
echo "Usage: create_filesystem <device> <filesystem_type> [label]"
echo "Supported types: ext4, xfs, btrfs"
return 1
fi
echo -e "${YELLOW}Creating $fstype filesystem on $device${NC}"
case "$fstype" in
"ext4")
if [ -n "$label" ]; then
mkfs.ext4 -L "$label" -m 1 "$device"
else
mkfs.ext4 -m 1 "$device"
fi
# Optimize for SSD if detected
if [[ $(cat /sys/block/$(basename $device | sed 's/[0-9]*$//')/queue/rotational) == "0" ]]; then
tune2fs -o discard "$device"
echo "SSD optimizations applied"
fi
;;
"xfs")
if [ -n "$label" ]; then
mkfs.xfs -f -L "$label" "$device"
else
mkfs.xfs -f "$device"
fi
;;
"btrfs")
if [ -n "$label" ]; then
mkfs.btrfs -f -L "$label" "$device"
else
mkfs.btrfs -f "$device"
fi
;;
*)
echo -e "${RED}Unsupported filesystem type: $fstype${NC}"
return 1
;;
esac
log_message "Created $fstype filesystem on $device"
echo -e "${GREEN}Filesystem created successfully${NC}"
}
# Mount filesystem with optimal options
mount_filesystem() {
local device="$1"
local mountpoint="$2"
local fstype="$3"
if [ -z "$device" ] || [ -z "$mountpoint" ]; then
echo "Usage: mount_filesystem <device> <mountpoint> [filesystem_type]"
return 1
fi
# Create mountpoint if it doesn't exist
mkdir -p "$mountpoint"
# Detect filesystem type if not provided
if [ -z "$fstype" ]; then
fstype=$(blkid -o value -s TYPE "$device")
fi
# Set optimal mount options based on filesystem type
case "$fstype" in
"ext4")
mount_options="defaults,noatime"
# Add discard for SSDs
if [[ $(cat /sys/block/$(basename $device | sed 's/[0-9]*$//')/queue/rotational) == "0" ]]; then
mount_options="$mount_options,discard"
fi
;;
"xfs")
mount_options="defaults,noatime"
;;
"btrfs")
mount_options="defaults,noatime,compress=zstd"
;;
*)
mount_options="defaults"
;;
esac
echo -e "${YELLOW}Mounting $device at $mountpoint with options: $mount_options${NC}"
if mount -o "$mount_options" "$device" "$mountpoint"; then
log_message "Mounted $device at $mountpoint"
echo -e "${GREEN}Filesystem mounted successfully${NC}"
# Add to fstab for permanent mounting
uuid=$(blkid -o value -s UUID "$device")
if [ -n "$uuid" ]; then
echo "UUID=$uuid $mountpoint $fstype $mount_options 0 2" >> /etc/fstab
echo "Added to /etc/fstab for automatic mounting"
fi
else
echo -e "${RED}Failed to mount filesystem${NC}"
return 1
fi
}
# Check filesystem health
check_filesystem_health() {
local device="$1"
if [ -z "$device" ]; then
echo "Usage: check_filesystem_health <device>"
return 1
fi
echo -e "${BLUE}=== Checking filesystem health for $device ===${NC}"
# Get filesystem type
fstype=$(blkid -o value -s TYPE "$device")
case "$fstype" in
"ext4"|"ext3"|"ext2")
echo "Running fsck for ext filesystem..."
fsck.ext4 -n "$device"
;;
"xfs")
echo "Running xfs_check..."
xfs_check "$device"
;;
"btrfs")
echo "Running btrfs check..."
btrfs check "$device"
;;
*)
echo "Running generic fsck..."
fsck -n "$device"
;;
esac
log_message "Filesystem health check completed for $device"
}
# Optimize filesystem performance
optimize_filesystem() {
local mountpoint="$1"
if [ -z "$mountpoint" ]; then
echo "Usage: optimize_filesystem <mountpoint>"
return 1
fi
echo -e "${BLUE}=== Optimizing filesystem at $mountpoint ===${NC}"
# Get device and filesystem type
device=$(df "$mountpoint" | tail -1 | awk '{print $1}')
fstype=$(df -T "$mountpoint" | tail -1 | awk '{print $2}')
case "$fstype" in
"ext4")
echo "Optimizing ext4 filesystem..."
# Defragment if needed
e4defrag -c "$mountpoint"
# Optimize inode allocation
tune2fs -o journal_data_writeback "$device"
# Enable TRIM for SSDs
if [[ $(cat /sys/block/$(basename $device | sed 's/[0-9]*$//')/queue/rotational) == "0" ]]; then
tune2fs -o discard "$device"
echo "SSD optimizations applied"
fi
;;
"xfs")
echo "Optimizing XFS filesystem..."
# XFS is generally well-optimized by default
# Check if online defragmentation is needed
xfs_fsr -v "$mountpoint"
;;
"btrfs")
echo "Optimizing Btrfs filesystem..."
# Balance filesystem
btrfs balance start "$mountpoint"
# Defragment
btrfs filesystem defragment -r "$mountpoint"
;;
esac
log_message "Filesystem optimization completed for $mountpoint"
echo -e "${GREEN}Optimization completed${NC}"
}
# Backup filesystem
backup_filesystem() {
local source="$1"
local backup_name="$2"
if [ -z "$source" ]; then
echo "Usage: backup_filesystem <source_mountpoint> [backup_name]"
return 1
fi
if [ -z "$backup_name" ]; then
backup_name="filesystem_backup_$(date +%Y%m%d_%H%M%S)"
fi
backup_path="$BACKUP_DIR/$backup_name"
echo -e "${YELLOW}Creating backup of $source to $backup_path${NC}"
# Create backup directory
mkdir -p "$backup_path"
# Use rsync for efficient backup
if rsync -avH --progress "$source/" "$backup_path/"; then
log_message "Backup created: $source -> $backup_path"
echo -e "${GREEN}Backup completed successfully${NC}"
# Create metadata file
cat > "$backup_path/.backup_info" << EOF
Source: $source
Backup Date: $(date)
Backup Size: $(du -sh "$backup_path" | cut -f1)
EOF
else
echo -e "${RED}Backup failed${NC}"
return 1
fi
}
# Monitor filesystem usage
monitor_filesystem() {
echo -e "${BLUE}=== Filesystem Usage Monitor ===${NC}"
while true; do
clear
echo "Filesystem Usage (updated every 5 seconds) - Press Ctrl+C to exit"
echo "================================================================"
df -h | grep -E "^Filesystem|^/dev"
echo ""
echo "I/O Statistics:"
iostat -x 1 1 | tail -n +4
echo ""
echo "Top 5 largest directories in /:"
du -h --max-depth=1 / 2>/dev/null | sort -hr | head -5
sleep 5
done
}
# Main menu
case "${1:-help}" in
"info")
show_filesystem_info
;;
"create")
check_root
create_filesystem "$2" "$3" "$4"
;;
"mount")
check_root
mount_filesystem "$2" "$3" "$4"
;;
"check")
check_root
check_filesystem_health "$2"
;;
"optimize")
check_root
optimize_filesystem "$2"
;;
"backup")
check_root
backup_filesystem "$2" "$3"
;;
"monitor")
monitor_filesystem
;;
"help"|*)
echo "Filesystem Manager for Ubuntu 22.04"
echo "Usage: $0 [command] [options]"
echo ""
echo "Commands:"
echo " info - Show filesystem information"
echo " create <device> <type> [label] - Create filesystem"
echo " mount <device> <mountpoint> - Mount filesystem with optimal options"
echo " check <device> - Check filesystem health"
echo " optimize <mountpoint> - Optimize filesystem performance"
echo " backup <source> [name] - Backup filesystem"
echo " monitor - Monitor filesystem usage"
echo ""
echo "Supported filesystem types: ext4, xfs, btrfs"
echo ""
echo "Examples:"
echo " $0 create /dev/sdb1 ext4 MyData"
echo " $0 mount /dev/sdb1 /mnt/data"
echo " $0 check /dev/sdb1"
echo " $0 optimize /home"
echo " $0 backup /home home_backup"
;;
esac
Best Practices for File Systems
Choosing the Right Filesystem
Decision Matrix:
Use Case | Recommended Filesystem
----------------------------|----------------------
General desktop/server | ext4
Large files/databases | XFS
Snapshots/modern features | Btrfs
Maximum data integrity | ZFS
Cross-platform compatibility| FAT32/NTFS
High-performance computing | XFS or ext4
Performance Optimization
Mount Options:
# For general use
mount -o defaults,noatime /dev/sdb1 /mnt/data
# For SSDs
mount -o defaults,noatime,discard /dev/sdb1 /mnt/data
# For databases
mount -o defaults,noatime,barrier=0 /dev/sdb1 /mnt/database
I/O Scheduler:
# For SSDs
echo none > /sys/block/sda/queue/scheduler
# For HDDs
echo mq-deadline > /sys/block/sda/queue/scheduler
Filesystem Tuning:
# Optimize ext4 for SSDs
tune2fs -o discard /dev/sda1
# Reduce reserved space (ext4)
tune2fs -m 1 /dev/sda1
Security and Maintenance
Regular Checks:
# Schedule regular filesystem checks
echo "0 3 * * 0 /sbin/fsck -A -f" >> /etc/crontab
Backup Strategy:
# Automated backups
rsync -avH /home/ /backup/home/
# Filesystem snapshots (Btrfs)
btrfs subvolume snapshot /home /home/.snapshots/$(date +%Y%m%d)
Monitoring:
# Set up disk usage alerts
echo "df -h | awk 'NR>1 {if(\$5+0 > 85) print \$0}' | mail -s 'Disk Alert' admin@example.com" | crontab -
File Systems — Storage Systems and Data Management Guide 1.0 documentation
File Systems
Understanding File Systems
A file system is a method used by operating systems to store, organize, and manage files on storage devices. It defines how data is stored, accessed, and organized on disks.
What is a File System?
A file system provides:
File Organization: Hierarchical structure for organizing files and directories
Metadata Management: Information about files (size, permissions, timestamps)
Space Management: Allocation and deallocation of disk space
Access Control: Security and permission mechanisms
Data Integrity: Protection against data corruption
File System Components
Core Components
Superblock: Contains metadata about the filesystem
Inode Table: Stores file metadata and pointers to data blocks
Data Blocks: Actual file content storage
Directory Structure: Hierarchical organization of files and folders
Journal: Transaction log for filesystem changes (in journaling filesystems)
File System Structure
Filesystem Layout
├── Superblock (filesystem metadata)
├── Group Descriptors (block group information)
├── Block Bitmap (free/used block tracking)
├── Inode Bitmap (free/used inode tracking)
├── Inode Table (file metadata)
└── Data Blocks (actual file content)
Common File Systems in Ubuntu 22.04
ext4 (Fourth Extended Filesystem)
The default filesystem for Ubuntu 22.04:
Features:
* Journaling for data integrity
* Large file and filesystem support (up to 1 EB)
* Backward compatibility with ext2/ext3
* Online defragmentation
* Delayed allocation
Use Cases:
* General purpose computing
* Desktop and server installations
* Boot partitions
* Home directories
# Create ext4 filesystem
sudo mkfs.ext4 /dev/sdb1
# Check ext4 filesystem
sudo fsck.ext4 /dev/sdb1
# Get ext4 filesystem information
sudo tune2fs -l /dev/sdb1
# Optimize ext4 filesystem
sudo tune2fs -o journal_data_writeback /dev/sdb1
XFS (eXtended File System)
High-performance 64-bit journaling filesystem:
Features:
* Excellent scalability
* Online resizing (grow only)
* Advanced quota management
* Allocation groups for parallel I/O
* Metadata journaling
Use Cases:
* Large files and databases
* High-performance computing
* Video editing and media storage
* Enterprise storage systems
# Create XFS filesystem
sudo mkfs.xfs /dev/sdb1
# Check XFS filesystem
sudo xfs_check /dev/sdb1
# Repair XFS filesystem
sudo xfs_repair /dev/sdb1
# Get XFS information
sudo xfs_info /dev/sdb1
# Resize XFS filesystem (grow only)
sudo xfs_growfs /mnt/xfs
Btrfs (B-tree File System)
Modern copy-on-write filesystem:
Features:
* Snapshots and cloning
* Built-in RAID support
* Compression (zlib, lzo, zstd)
* Checksumming for data integrity
* Online resizing (grow and shrink)
Use Cases:
* System snapshots
* Development environments
* Data deduplication scenarios
* Advanced storage management
# Create Btrfs filesystem
sudo mkfs.btrfs /dev/sdb1
# Mount with compression
sudo mount -o compress=zstd /dev/sdb1 /mnt/btrfs
# Create snapshot
sudo btrfs subvolume snapshot /mnt/btrfs /mnt/btrfs/snapshot
# List subvolumes
sudo btrfs subvolume list /mnt/btrfs
ZFS (Zettabyte File System)
Advanced filesystem with built-in volume management:
Features:
* Built-in RAID (RAID-Z)
* Snapshots and clones
* Data deduplication
* Compression
* End-to-end checksumming
Installation and Use:
# Install ZFS on Ubuntu 22.04
sudo apt update
sudo apt install zfsutils-linux
# Create ZFS pool
sudo zpool create mypool /dev/sdb
# Create ZFS dataset
sudo zfs create mypool/data
# Enable compression
sudo zfs set compression=lz4 mypool/data
# Create snapshot
sudo zfs snapshot mypool/data@snapshot1
FAT32 and NTFS
FAT32: Universal compatibility, limited file size (4GB max)
NTFS: Windows filesystem with advanced features
# Create FAT32 filesystem
sudo mkfs.fat -F32 /dev/sdb1
# Create NTFS filesystem
sudo mkfs.ntfs /dev/sdb1
# Mount NTFS with full permissions
sudo mount -t ntfs-3g /dev/sdb1 /mnt/ntfs -o permissions
File System Operations in Ubuntu 22.04
Creating File Systems
# ext4 with custom options
sudo mkfs.ext4 -L "MyData" -m 1 /dev/sdb1
# XFS with custom block size
sudo mkfs.xfs -f -b size=4096 /dev/sdb1
# Btrfs with multiple devices (RAID)
sudo mkfs.btrfs -d raid1 -m raid1 /dev/sdb1 /dev/sdc1
# Set filesystem label
sudo e2label /dev/sdb1 "DataDisk"
Mounting File Systems
# Basic mounting
sudo mount /dev/sdb1 /mnt/data
# Mount with specific options
sudo mount -o rw,noatime,discard /dev/sdb1 /mnt/data
# Mount by UUID (preferred method)
sudo mount UUID=12345678-1234-1234-1234-123456789abc /mnt/data
# Mount with user permissions
sudo mount -o uid=1000,gid=1000 /dev/sdb1 /mnt/data
Checking and Repairing File Systems
# Check filesystem (read-only)
sudo fsck -n /dev/sdb1
# Automatic repair
sudo fsck -p /dev/sdb1
# Force check and repair
sudo fsck -f /dev/sdb1
# Check specific filesystem types
sudo fsck.ext4 /dev/sdb1
sudo xfs_check /dev/sdb1
sudo btrfs check /dev/sdb1
File System Monitoring
Monitoring Tools and Commands
# Real-time filesystem usage
watch df -h
# Inode usage
df -i
# Detailed filesystem information
stat -f /
# Monitor filesystem I/O
sudo iotop -o
# Check filesystem fragmentation (ext4)
sudo e4defrag -c /dev/sdb1
# Btrfs filesystem usage
sudo btrfs filesystem usage /mnt/btrfs
Frequently Asked Questions
Q: Which filesystem should I choose for Ubuntu 22.04?
A: Filesystem selection depends on your use case:
ext4: Best general-purpose choice, default for Ubuntu
XFS: Large files, databases, high-performance needs
Btrfs: Need snapshots, compression, or advanced features
ZFS: Enterprise features, data integrity critical
# For most users (recommended)
sudo mkfs.ext4 /dev/sdb1
# For large files and databases
sudo mkfs.xfs /dev/sdb1
# For snapshots and modern features
sudo mkfs.btrfs /dev/sdb1
Q: How do I convert between filesystems?
A: Filesystem conversion requires backup and restore:
# 1. Backup data
sudo rsync -av /mnt/source/ /backup/
# 2. Unmount and recreate filesystem
sudo umount /mnt/source
sudo mkfs.ext4 /dev/sdb1
# 3. Mount and restore data
sudo mount /dev/sdb1 /mnt/source
sudo rsync -av /backup/ /mnt/source/
Q: How do I optimize filesystem performance?
A: Several optimization techniques:
# For SSDs - enable TRIM
sudo mount -o discard /dev/sdb1 /mnt/data
# Disable access time updates
sudo mount -o noatime /dev/sdb1 /mnt/data
# For databases - use direct I/O
sudo mount -o barrier=0 /dev/sdb1 /mnt/database
# Optimize ext4 for SSDs
sudo tune2fs -o discard /dev/sdb1
Q: How do I recover deleted files?
A: Recovery methods depend on the filesystem:
# Install recovery tools
sudo apt install testdisk photorec extundelete
# For ext4 filesystems
sudo extundelete /dev/sdb1 --restore-all
# General purpose recovery
sudo photorec /dev/sdb1
# For immediate action after deletion
sudo grep -a -B25 -A25 'text from deleted file' /dev/sdb1
Coding Examples
Python Filesystem Analyzer
#!/usr/bin/env python3
"""
Filesystem analyzer for Ubuntu 22.04
"""
import os
import subprocess
import json
import time
from pathlib import Path
class FilesystemAnalyzer:
def __init__(self):
self.mountpoints = self.get_mountpoints()
def get_mountpoints(self):
"""Get all mounted filesystems"""
mountpoints = []
try:
with open('/proc/mounts', 'r') as f:
for line in f:
parts = line.strip().split()
if len(parts) >= 3 and parts[0].startswith('/dev/'):
mountpoints.append({
'device': parts[0],
'mountpoint': parts[1],
'filesystem': parts[2],
'options': parts[3]
})
except Exception as e:
print(f"Error reading mount points: {e}")
return mountpoints
def analyze_filesystem(self, path='/'):
"""Analyze filesystem usage and characteristics"""
try:
# Get basic filesystem statistics
statvfs = os.statvfs(path)
total_size = statvfs.f_frsize * statvfs.f_blocks
free_size = statvfs.f_frsize * statvfs.f_bavail
used_size = total_size - free_size
# Get inode information
total_inodes = statvfs.f_files
free_inodes = statvfs.f_favail
used_inodes = total_inodes - free_inodes
return {
'path': path,
'total_size_gb': round(total_size / (1024**3), 2),
'used_size_gb': round(used_size / (1024**3), 2),
'free_size_gb': round(free_size / (1024**3), 2),
'usage_percent': round((used_size / total_size) * 100, 2),
'total_inodes': total_inodes,
'used_inodes': used_inodes,
'free_inodes': free_inodes,
'inode_usage_percent': round((used_inodes / total_inodes) * 100, 2) if total_inodes > 0 else 0
}
except Exception as e:
return {'error': str(e)}
def get_largest_files(self, path='/', min_size_mb=100, count=10):
"""Find largest files in a directory tree"""
large_files = []
min_size_bytes = min_size_mb * 1024 * 1024
try:
for root, dirs, files in os.walk(path):
# Skip certain directories to avoid permission errors
dirs[:] = [d for d in dirs if not d.startswith('.') and d not in ['proc', 'sys', 'dev']]
for file in files:
try:
file_path = os.path.join(root, file)
file_size = os.path.getsize(file_path)
if file_size >= min_size_bytes:
large_files.append({
'path': file_path,
'size_mb': round(file_size / (1024**2), 2),
'size_gb': round(file_size / (1024**3), 2)
})
except (OSError, IOError):
continue # Skip files we can't access
# Sort by size and return top files
large_files.sort(key=lambda x: x['size_mb'], reverse=True)
return large_files[:count]
except Exception as e:
return [{'error': str(e)}]
def get_directory_sizes(self, path='/', max_depth=2):
"""Get sizes of directories"""
directory_sizes = []
try:
result = subprocess.run(['du', '-h', f'--max-depth={max_depth}', path],
capture_output=True, text=True)
for line in result.stdout.strip().split('\n'):
if line:
parts = line.split('\t')
if len(parts) == 2:
size, dir_path = parts
directory_sizes.append({
'path': dir_path,
'size': size
})
return directory_sizes
except Exception as e:
return [{'error': str(e)}]
def check_filesystem_health(self, device):
"""Check filesystem health using fsck"""
try:
# Run read-only check
result = subprocess.run(['fsck', '-n', device],
capture_output=True, text=True)
return {
'device': device,
'status': 'clean' if result.returncode == 0 else 'errors_found',
'output': result.stdout,
'errors': result.stderr
}
except Exception as e:
return {'device': device, 'error': str(e)}
def monitor_filesystem_performance(self, duration=10):
"""Monitor filesystem I/O performance"""
try:
# Start iostat monitoring
process = subprocess.Popen(['iostat', '-x', '1', str(duration)],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True)
output, error = process.communicate()
return {
'duration': duration,
'iostat_output': output,
'error': error if error else None
}
except Exception as e:
return {'error': str(e)}
def generate_report(self):
"""Generate comprehensive filesystem report"""
report = {
'timestamp': time.strftime('%Y-%m-%d %H:%M:%S'),
'hostname': os.uname().nodename,
'mountpoints': self.mountpoints,
'filesystem_analysis': [],
'large_files': [],
'directory_sizes': []
}
# Analyze each mounted filesystem
for mount in self.mountpoints:
analysis = self.analyze_filesystem(mount['mountpoint'])
analysis['device'] = mount['device']
analysis['filesystem_type'] = mount['filesystem']
report['filesystem_analysis'].append(analysis)
# Find large files in home and var directories
for path in ['/home', '/var']:
if os.path.exists(path):
large_files = self.get_largest_files(path, min_size_mb=50, count=5)
report['large_files'].extend(large_files)
# Get directory sizes for common directories
for path in ['/', '/home', '/var', '/usr']:
if os.path.exists(path):
dir_sizes = self.get_directory_sizes(path, max_depth=2)
report['directory_sizes'].extend(dir_sizes)
return report
# Example usage
if __name__ == "__main__":
analyzer = FilesystemAnalyzer()
# Generate and display report
report = analyzer.generate_report()
print(json.dumps(report, indent=2))
# Save report to file
with open(f'/tmp/filesystem_report_{int(time.time())}.json', 'w') as f:
json.dump(report, f, indent=2)
Bash Filesystem Management Script
#!/bin/bash
# filesystem_manager.sh - Comprehensive filesystem management for Ubuntu 22.04
# Configuration
BACKUP_DIR="/backup"
LOG_FILE="/var/log/filesystem_manager.log"
# Colors
RED='[0;31m'
GREEN='[0;32m'
YELLOW='[1;33m'
BLUE='[0;34m'
NC='[0m'
# Logging function
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$LOG_FILE"
}
# Check if running as root
check_root() {
if [[ $EUID -ne 0 ]]; then
echo -e "${RED}This script must be run as root${NC}"
exit 1
fi
}
# Display filesystem information
show_filesystem_info() {
echo -e "${BLUE}=== Filesystem Information ===${NC}"
echo "Mounted Filesystems:"
df -hT | grep -v tmpfs
echo ""
echo "Filesystem Types:"
lsblk -f
echo ""
echo "Mount Points:"
mount | grep "^/dev" | column -t
echo ""
}
# Create filesystem with optimal settings
create_filesystem() {
local device="$1"
local fstype="$2"
local label="$3"
if [ -z "$device" ] || [ -z "$fstype" ]; then
echo "Usage: create_filesystem <device> <filesystem_type> [label]"
echo "Supported types: ext4, xfs, btrfs"
return 1
fi
echo -e "${YELLOW}Creating $fstype filesystem on $device${NC}"
case "$fstype" in
"ext4")
if [ -n "$label" ]; then
mkfs.ext4 -L "$label" -m 1 "$device"
else
mkfs.ext4 -m 1 "$device"
fi
# Optimize for SSD if detected
if [[ $(cat /sys/block/$(basename $device | sed 's/[0-9]*$//')/queue/rotational) == "0" ]]; then
tune2fs -o discard "$device"
echo "SSD optimizations applied"
fi
;;
"xfs")
if [ -n "$label" ]; then
mkfs.xfs -f -L "$label" "$device"
else
mkfs.xfs -f "$device"
fi
;;
"btrfs")
if [ -n "$label" ]; then
mkfs.btrfs -f -L "$label" "$device"
else
mkfs.btrfs -f "$device"
fi
;;
*)
echo -e "${RED}Unsupported filesystem type: $fstype${NC}"
return 1
;;
esac
log_message "Created $fstype filesystem on $device"
echo -e "${GREEN}Filesystem created successfully${NC}"
}
# Mount filesystem with optimal options
mount_filesystem() {
local device="$1"
local mountpoint="$2"
local fstype="$3"
if [ -z "$device" ] || [ -z "$mountpoint" ]; then
echo "Usage: mount_filesystem <device> <mountpoint> [filesystem_type]"
return 1
fi
# Create mountpoint if it doesn't exist
mkdir -p "$mountpoint"
# Detect filesystem type if not provided
if [ -z "$fstype" ]; then
fstype=$(blkid -o value -s TYPE "$device")
fi
# Set optimal mount options based on filesystem type
case "$fstype" in
"ext4")
mount_options="defaults,noatime"
# Add discard for SSDs
if [[ $(cat /sys/block/$(basename $device | sed 's/[0-9]*$//')/queue/rotational) == "0" ]]; then
mount_options="$mount_options,discard"
fi
;;
"xfs")
mount_options="defaults,noatime"
;;
"btrfs")
mount_options="defaults,noatime,compress=zstd"
;;
*)
mount_options="defaults"
;;
esac
echo -e "${YELLOW}Mounting $device at $mountpoint with options: $mount_options${NC}"
if mount -o "$mount_options" "$device" "$mountpoint"; then
log_message "Mounted $device at $mountpoint"
echo -e "${GREEN}Filesystem mounted successfully${NC}"
# Add to fstab for permanent mounting
uuid=$(blkid -o value -s UUID "$device")
if [ -n "$uuid" ]; then
echo "UUID=$uuid $mountpoint $fstype $mount_options 0 2" >> /etc/fstab
echo "Added to /etc/fstab for automatic mounting"
fi
else
echo -e "${RED}Failed to mount filesystem${NC}"
return 1
fi
}
# Check filesystem health
check_filesystem_health() {
local device="$1"
if [ -z "$device" ]; then
echo "Usage: check_filesystem_health <device>"
return 1
fi
echo -e "${BLUE}=== Checking filesystem health for $device ===${NC}"
# Get filesystem type
fstype=$(blkid -o value -s TYPE "$device")
case "$fstype" in
"ext4"|"ext3"|"ext2")
echo "Running fsck for ext filesystem..."
fsck.ext4 -n "$device"
;;
"xfs")
echo "Running xfs_check..."
xfs_check "$device"
;;
"btrfs")
echo "Running btrfs check..."
btrfs check "$device"
;;
*)
echo "Running generic fsck..."
fsck -n "$device"
;;
esac
log_message "Filesystem health check completed for $device"
}
# Optimize filesystem performance
optimize_filesystem() {
local mountpoint="$1"
if [ -z "$mountpoint" ]; then
echo "Usage: optimize_filesystem <mountpoint>"
return 1
fi
echo -e "${BLUE}=== Optimizing filesystem at $mountpoint ===${NC}"
# Get device and filesystem type
device=$(df "$mountpoint" | tail -1 | awk '{print $1}')
fstype=$(df -T "$mountpoint" | tail -1 | awk '{print $2}')
case "$fstype" in
"ext4")
echo "Optimizing ext4 filesystem..."
# Defragment if needed
e4defrag -c "$mountpoint"
# Optimize inode allocation
tune2fs -o journal_data_writeback "$device"
# Enable TRIM for SSDs
if [[ $(cat /sys/block/$(basename $device | sed 's/[0-9]*$//')/queue/rotational) == "0" ]]; then
tune2fs -o discard "$device"
echo "SSD optimizations applied"
fi
;;
"xfs")
echo "Optimizing XFS filesystem..."
# XFS is generally well-optimized by default
# Check if online defragmentation is needed
xfs_fsr -v "$mountpoint"
;;
"btrfs")
echo "Optimizing Btrfs filesystem..."
# Balance filesystem
btrfs balance start "$mountpoint"
# Defragment
btrfs filesystem defragment -r "$mountpoint"
;;
esac
log_message "Filesystem optimization completed for $mountpoint"
echo -e "${GREEN}Optimization completed${NC}"
}
# Backup filesystem
backup_filesystem() {
local source="$1"
local backup_name="$2"
if [ -z "$source" ]; then
echo "Usage: backup_filesystem <source_mountpoint> [backup_name]"
return 1
fi
if [ -z "$backup_name" ]; then
backup_name="filesystem_backup_$(date +%Y%m%d_%H%M%S)"
fi
backup_path="$BACKUP_DIR/$backup_name"
echo -e "${YELLOW}Creating backup of $source to $backup_path${NC}"
# Create backup directory
mkdir -p "$backup_path"
# Use rsync for efficient backup
if rsync -avH --progress "$source/" "$backup_path/"; then
log_message "Backup created: $source -> $backup_path"
echo -e "${GREEN}Backup completed successfully${NC}"
# Create metadata file
cat > "$backup_path/.backup_info" << EOF
Source: $source
Backup Date: $(date)
Backup Size: $(du -sh "$backup_path" | cut -f1)
EOF
else
echo -e "${RED}Backup failed${NC}"
return 1
fi
}
# Monitor filesystem usage
monitor_filesystem() {
echo -e "${BLUE}=== Filesystem Usage Monitor ===${NC}"
while true; do
clear
echo "Filesystem Usage (updated every 5 seconds) - Press Ctrl+C to exit"
echo "================================================================"
df -h | grep -E "^Filesystem|^/dev"
echo ""
echo "I/O Statistics:"
iostat -x 1 1 | tail -n +4
echo ""
echo "Top 5 largest directories in /:"
du -h --max-depth=1 / 2>/dev/null | sort -hr | head -5
sleep 5
done
}
# Main menu
case "${1:-help}" in
"info")
show_filesystem_info
;;
"create")
check_root
create_filesystem "$2" "$3" "$4"
;;
"mount")
check_root
mount_filesystem "$2" "$3" "$4"
;;
"check")
check_root
check_filesystem_health "$2"
;;
"optimize")
check_root
optimize_filesystem "$2"
;;
"backup")
check_root
backup_filesystem "$2" "$3"
;;
"monitor")
monitor_filesystem
;;
"help"|*)
echo "Filesystem Manager for Ubuntu 22.04"
echo "Usage: $0 [command] [options]"
echo ""
echo "Commands:"
echo " info - Show filesystem information"
echo " create <device> <type> [label] - Create filesystem"
echo " mount <device> <mountpoint> - Mount filesystem with optimal options"
echo " check <device> - Check filesystem health"
echo " optimize <mountpoint> - Optimize filesystem performance"
echo " backup <source> [name] - Backup filesystem"
echo " monitor - Monitor filesystem usage"
echo ""
echo "Supported filesystem types: ext4, xfs, btrfs"
echo ""
echo "Examples:"
echo " $0 create /dev/sdb1 ext4 MyData"
echo " $0 mount /dev/sdb1 /mnt/data"
echo " $0 check /dev/sdb1"
echo " $0 optimize /home"
echo " $0 backup /home home_backup"
;;
esac
Best Practices for File Systems
Choosing the Right Filesystem
Decision Matrix:
Use Case | Recommended Filesystem
----------------------------|----------------------
General desktop/server | ext4
Large files/databases | XFS
Snapshots/modern features | Btrfs
Maximum data integrity | ZFS
Cross-platform compatibility| FAT32/NTFS
High-performance computing | XFS or ext4
Performance Optimization
Mount Options:
# For general use
mount -o defaults,noatime /dev/sdb1 /mnt/data
# For SSDs
mount -o defaults,noatime,discard /dev/sdb1 /mnt/data
# For databases
mount -o defaults,noatime,barrier=0 /dev/sdb1 /mnt/database
I/O Scheduler:
# For SSDs
echo none > /sys/block/sda/queue/scheduler
# For HDDs
echo mq-deadline > /sys/block/sda/queue/scheduler
Filesystem Tuning:
# Optimize ext4 for SSDs
tune2fs -o discard /dev/sda1
# Reduce reserved space (ext4)
tune2fs -m 1 /dev/sda1
Security and Maintenance
Regular Checks:
# Schedule regular filesystem checks
echo "0 3 * * 0 /sbin/fsck -A -f" >> /etc/crontab
Backup Strategy:
# Automated backups
rsync -avH /home/ /backup/home/
# Filesystem snapshots (Btrfs)
btrfs subvolume snapshot /home /home/.snapshots/$(date +%Y%m%d)
Monitoring:
# Set up disk usage alerts
echo "df -h | awk 'NR>1 {if(\$5+0 > 85) print \$0}' | mail -s 'Disk Alert' admin@example.com" | crontab -
File Systems
Understanding File Systems
A file system is a method used by operating systems to store, organize, and manage files on storage devices. It defines how data is stored, accessed, and organized on disks.
What is a File System?
A file system provides:
File Organization: Hierarchical structure for organizing files and directories
Metadata Management: Information about files (size, permissions, timestamps)
Space Management: Allocation and deallocation of disk space
Access Control: Security and permission mechanisms
Data Integrity: Protection against data corruption
File System Components
Core Components
Superblock: Contains metadata about the filesystem
Inode Table: Stores file metadata and pointers to data blocks
Data Blocks: Actual file content storage
Directory Structure: Hierarchical organization of files and folders
Journal: Transaction log for filesystem changes (in journaling filesystems)
File System Structure
Filesystem Layout
├── Superblock (filesystem metadata)
├── Group Descriptors (block group information)
├── Block Bitmap (free/used block tracking)
├── Inode Bitmap (free/used inode tracking)
├── Inode Table (file metadata)
└── Data Blocks (actual file content)
Common File Systems in Ubuntu 22.04
ext4 (Fourth Extended Filesystem)
The default filesystem for Ubuntu 22.04:
Features: * Journaling for data integrity * Large file and filesystem support (up to 1 EB) * Backward compatibility with ext2/ext3 * Online defragmentation * Delayed allocation
Use Cases: * General purpose computing * Desktop and server installations * Boot partitions * Home directories
# Create ext4 filesystem
sudo mkfs.ext4 /dev/sdb1
# Check ext4 filesystem
sudo fsck.ext4 /dev/sdb1
# Get ext4 filesystem information
sudo tune2fs -l /dev/sdb1
# Optimize ext4 filesystem
sudo tune2fs -o journal_data_writeback /dev/sdb1
XFS (eXtended File System)
High-performance 64-bit journaling filesystem:
Features: * Excellent scalability * Online resizing (grow only) * Advanced quota management * Allocation groups for parallel I/O * Metadata journaling
Use Cases: * Large files and databases * High-performance computing * Video editing and media storage * Enterprise storage systems
# Create XFS filesystem
sudo mkfs.xfs /dev/sdb1
# Check XFS filesystem
sudo xfs_check /dev/sdb1
# Repair XFS filesystem
sudo xfs_repair /dev/sdb1
# Get XFS information
sudo xfs_info /dev/sdb1
# Resize XFS filesystem (grow only)
sudo xfs_growfs /mnt/xfs
Btrfs (B-tree File System)
Modern copy-on-write filesystem:
Features: * Snapshots and cloning * Built-in RAID support * Compression (zlib, lzo, zstd) * Checksumming for data integrity * Online resizing (grow and shrink)
Use Cases: * System snapshots * Development environments * Data deduplication scenarios * Advanced storage management
# Create Btrfs filesystem
sudo mkfs.btrfs /dev/sdb1
# Mount with compression
sudo mount -o compress=zstd /dev/sdb1 /mnt/btrfs
# Create snapshot
sudo btrfs subvolume snapshot /mnt/btrfs /mnt/btrfs/snapshot
# List subvolumes
sudo btrfs subvolume list /mnt/btrfs
ZFS (Zettabyte File System)
Advanced filesystem with built-in volume management:
Features: * Built-in RAID (RAID-Z) * Snapshots and clones * Data deduplication * Compression * End-to-end checksumming
Installation and Use:
# Install ZFS on Ubuntu 22.04
sudo apt update
sudo apt install zfsutils-linux
# Create ZFS pool
sudo zpool create mypool /dev/sdb
# Create ZFS dataset
sudo zfs create mypool/data
# Enable compression
sudo zfs set compression=lz4 mypool/data
# Create snapshot
sudo zfs snapshot mypool/data@snapshot1
FAT32 and NTFS
FAT32: Universal compatibility, limited file size (4GB max) NTFS: Windows filesystem with advanced features
# Create FAT32 filesystem
sudo mkfs.fat -F32 /dev/sdb1
# Create NTFS filesystem
sudo mkfs.ntfs /dev/sdb1
# Mount NTFS with full permissions
sudo mount -t ntfs-3g /dev/sdb1 /mnt/ntfs -o permissions
File System Operations in Ubuntu 22.04
Creating File Systems
# ext4 with custom options
sudo mkfs.ext4 -L "MyData" -m 1 /dev/sdb1
# XFS with custom block size
sudo mkfs.xfs -f -b size=4096 /dev/sdb1
# Btrfs with multiple devices (RAID)
sudo mkfs.btrfs -d raid1 -m raid1 /dev/sdb1 /dev/sdc1
# Set filesystem label
sudo e2label /dev/sdb1 "DataDisk"
Mounting File Systems
# Basic mounting
sudo mount /dev/sdb1 /mnt/data
# Mount with specific options
sudo mount -o rw,noatime,discard /dev/sdb1 /mnt/data
# Mount by UUID (preferred method)
sudo mount UUID=12345678-1234-1234-1234-123456789abc /mnt/data
# Mount with user permissions
sudo mount -o uid=1000,gid=1000 /dev/sdb1 /mnt/data
Checking and Repairing File Systems
# Check filesystem (read-only)
sudo fsck -n /dev/sdb1
# Automatic repair
sudo fsck -p /dev/sdb1
# Force check and repair
sudo fsck -f /dev/sdb1
# Check specific filesystem types
sudo fsck.ext4 /dev/sdb1
sudo xfs_check /dev/sdb1
sudo btrfs check /dev/sdb1
File System Monitoring
Monitoring Tools and Commands
# Real-time filesystem usage
watch df -h
# Inode usage
df -i
# Detailed filesystem information
stat -f /
# Monitor filesystem I/O
sudo iotop -o
# Check filesystem fragmentation (ext4)
sudo e4defrag -c /dev/sdb1
# Btrfs filesystem usage
sudo btrfs filesystem usage /mnt/btrfs
Frequently Asked Questions
Q: Which filesystem should I choose for Ubuntu 22.04?
A: Filesystem selection depends on your use case:
ext4: Best general-purpose choice, default for Ubuntu
XFS: Large files, databases, high-performance needs
Btrfs: Need snapshots, compression, or advanced features
ZFS: Enterprise features, data integrity critical
# For most users (recommended)
sudo mkfs.ext4 /dev/sdb1
# For large files and databases
sudo mkfs.xfs /dev/sdb1
# For snapshots and modern features
sudo mkfs.btrfs /dev/sdb1
Q: How do I convert between filesystems?
A: Filesystem conversion requires backup and restore:
# 1. Backup data
sudo rsync -av /mnt/source/ /backup/
# 2. Unmount and recreate filesystem
sudo umount /mnt/source
sudo mkfs.ext4 /dev/sdb1
# 3. Mount and restore data
sudo mount /dev/sdb1 /mnt/source
sudo rsync -av /backup/ /mnt/source/
Q: How do I optimize filesystem performance?
A: Several optimization techniques:
# For SSDs - enable TRIM
sudo mount -o discard /dev/sdb1 /mnt/data
# Disable access time updates
sudo mount -o noatime /dev/sdb1 /mnt/data
# For databases - use direct I/O
sudo mount -o barrier=0 /dev/sdb1 /mnt/database
# Optimize ext4 for SSDs
sudo tune2fs -o discard /dev/sdb1
Q: How do I recover deleted files?
A: Recovery methods depend on the filesystem:
# Install recovery tools
sudo apt install testdisk photorec extundelete
# For ext4 filesystems
sudo extundelete /dev/sdb1 --restore-all
# General purpose recovery
sudo photorec /dev/sdb1
# For immediate action after deletion
sudo grep -a -B25 -A25 'text from deleted file' /dev/sdb1
Coding Examples
Python Filesystem Analyzer
#!/usr/bin/env python3
"""
Filesystem analyzer for Ubuntu 22.04
"""
import os
import subprocess
import json
import time
from pathlib import Path
class FilesystemAnalyzer:
def __init__(self):
self.mountpoints = self.get_mountpoints()
def get_mountpoints(self):
"""Get all mounted filesystems"""
mountpoints = []
try:
with open('/proc/mounts', 'r') as f:
for line in f:
parts = line.strip().split()
if len(parts) >= 3 and parts[0].startswith('/dev/'):
mountpoints.append({
'device': parts[0],
'mountpoint': parts[1],
'filesystem': parts[2],
'options': parts[3]
})
except Exception as e:
print(f"Error reading mount points: {e}")
return mountpoints
def analyze_filesystem(self, path='/'):
"""Analyze filesystem usage and characteristics"""
try:
# Get basic filesystem statistics
statvfs = os.statvfs(path)
total_size = statvfs.f_frsize * statvfs.f_blocks
free_size = statvfs.f_frsize * statvfs.f_bavail
used_size = total_size - free_size
# Get inode information
total_inodes = statvfs.f_files
free_inodes = statvfs.f_favail
used_inodes = total_inodes - free_inodes
return {
'path': path,
'total_size_gb': round(total_size / (1024**3), 2),
'used_size_gb': round(used_size / (1024**3), 2),
'free_size_gb': round(free_size / (1024**3), 2),
'usage_percent': round((used_size / total_size) * 100, 2),
'total_inodes': total_inodes,
'used_inodes': used_inodes,
'free_inodes': free_inodes,
'inode_usage_percent': round((used_inodes / total_inodes) * 100, 2) if total_inodes > 0 else 0
}
except Exception as e:
return {'error': str(e)}
def get_largest_files(self, path='/', min_size_mb=100, count=10):
"""Find largest files in a directory tree"""
large_files = []
min_size_bytes = min_size_mb * 1024 * 1024
try:
for root, dirs, files in os.walk(path):
# Skip certain directories to avoid permission errors
dirs[:] = [d for d in dirs if not d.startswith('.') and d not in ['proc', 'sys', 'dev']]
for file in files:
try:
file_path = os.path.join(root, file)
file_size = os.path.getsize(file_path)
if file_size >= min_size_bytes:
large_files.append({
'path': file_path,
'size_mb': round(file_size / (1024**2), 2),
'size_gb': round(file_size / (1024**3), 2)
})
except (OSError, IOError):
continue # Skip files we can't access
# Sort by size and return top files
large_files.sort(key=lambda x: x['size_mb'], reverse=True)
return large_files[:count]
except Exception as e:
return [{'error': str(e)}]
def get_directory_sizes(self, path='/', max_depth=2):
"""Get sizes of directories"""
directory_sizes = []
try:
result = subprocess.run(['du', '-h', f'--max-depth={max_depth}', path],
capture_output=True, text=True)
for line in result.stdout.strip().split('\n'):
if line:
parts = line.split('\t')
if len(parts) == 2:
size, dir_path = parts
directory_sizes.append({
'path': dir_path,
'size': size
})
return directory_sizes
except Exception as e:
return [{'error': str(e)}]
def check_filesystem_health(self, device):
"""Check filesystem health using fsck"""
try:
# Run read-only check
result = subprocess.run(['fsck', '-n', device],
capture_output=True, text=True)
return {
'device': device,
'status': 'clean' if result.returncode == 0 else 'errors_found',
'output': result.stdout,
'errors': result.stderr
}
except Exception as e:
return {'device': device, 'error': str(e)}
def monitor_filesystem_performance(self, duration=10):
"""Monitor filesystem I/O performance"""
try:
# Start iostat monitoring
process = subprocess.Popen(['iostat', '-x', '1', str(duration)],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True)
output, error = process.communicate()
return {
'duration': duration,
'iostat_output': output,
'error': error if error else None
}
except Exception as e:
return {'error': str(e)}
def generate_report(self):
"""Generate comprehensive filesystem report"""
report = {
'timestamp': time.strftime('%Y-%m-%d %H:%M:%S'),
'hostname': os.uname().nodename,
'mountpoints': self.mountpoints,
'filesystem_analysis': [],
'large_files': [],
'directory_sizes': []
}
# Analyze each mounted filesystem
for mount in self.mountpoints:
analysis = self.analyze_filesystem(mount['mountpoint'])
analysis['device'] = mount['device']
analysis['filesystem_type'] = mount['filesystem']
report['filesystem_analysis'].append(analysis)
# Find large files in home and var directories
for path in ['/home', '/var']:
if os.path.exists(path):
large_files = self.get_largest_files(path, min_size_mb=50, count=5)
report['large_files'].extend(large_files)
# Get directory sizes for common directories
for path in ['/', '/home', '/var', '/usr']:
if os.path.exists(path):
dir_sizes = self.get_directory_sizes(path, max_depth=2)
report['directory_sizes'].extend(dir_sizes)
return report
# Example usage
if __name__ == "__main__":
analyzer = FilesystemAnalyzer()
# Generate and display report
report = analyzer.generate_report()
print(json.dumps(report, indent=2))
# Save report to file
with open(f'/tmp/filesystem_report_{int(time.time())}.json', 'w') as f:
json.dump(report, f, indent=2)
Bash Filesystem Management Script
#!/bin/bash
# filesystem_manager.sh - Comprehensive filesystem management for Ubuntu 22.04
# Configuration
BACKUP_DIR="/backup"
LOG_FILE="/var/log/filesystem_manager.log"
# Colors
RED='[0;31m'
GREEN='[0;32m'
YELLOW='[1;33m'
BLUE='[0;34m'
NC='[0m'
# Logging function
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$LOG_FILE"
}
# Check if running as root
check_root() {
if [[ $EUID -ne 0 ]]; then
echo -e "${RED}This script must be run as root${NC}"
exit 1
fi
}
# Display filesystem information
show_filesystem_info() {
echo -e "${BLUE}=== Filesystem Information ===${NC}"
echo "Mounted Filesystems:"
df -hT | grep -v tmpfs
echo ""
echo "Filesystem Types:"
lsblk -f
echo ""
echo "Mount Points:"
mount | grep "^/dev" | column -t
echo ""
}
# Create filesystem with optimal settings
create_filesystem() {
local device="$1"
local fstype="$2"
local label="$3"
if [ -z "$device" ] || [ -z "$fstype" ]; then
echo "Usage: create_filesystem <device> <filesystem_type> [label]"
echo "Supported types: ext4, xfs, btrfs"
return 1
fi
echo -e "${YELLOW}Creating $fstype filesystem on $device${NC}"
case "$fstype" in
"ext4")
if [ -n "$label" ]; then
mkfs.ext4 -L "$label" -m 1 "$device"
else
mkfs.ext4 -m 1 "$device"
fi
# Optimize for SSD if detected
if [[ $(cat /sys/block/$(basename $device | sed 's/[0-9]*$//')/queue/rotational) == "0" ]]; then
tune2fs -o discard "$device"
echo "SSD optimizations applied"
fi
;;
"xfs")
if [ -n "$label" ]; then
mkfs.xfs -f -L "$label" "$device"
else
mkfs.xfs -f "$device"
fi
;;
"btrfs")
if [ -n "$label" ]; then
mkfs.btrfs -f -L "$label" "$device"
else
mkfs.btrfs -f "$device"
fi
;;
*)
echo -e "${RED}Unsupported filesystem type: $fstype${NC}"
return 1
;;
esac
log_message "Created $fstype filesystem on $device"
echo -e "${GREEN}Filesystem created successfully${NC}"
}
# Mount filesystem with optimal options
mount_filesystem() {
local device="$1"
local mountpoint="$2"
local fstype="$3"
if [ -z "$device" ] || [ -z "$mountpoint" ]; then
echo "Usage: mount_filesystem <device> <mountpoint> [filesystem_type]"
return 1
fi
# Create mountpoint if it doesn't exist
mkdir -p "$mountpoint"
# Detect filesystem type if not provided
if [ -z "$fstype" ]; then
fstype=$(blkid -o value -s TYPE "$device")
fi
# Set optimal mount options based on filesystem type
case "$fstype" in
"ext4")
mount_options="defaults,noatime"
# Add discard for SSDs
if [[ $(cat /sys/block/$(basename $device | sed 's/[0-9]*$//')/queue/rotational) == "0" ]]; then
mount_options="$mount_options,discard"
fi
;;
"xfs")
mount_options="defaults,noatime"
;;
"btrfs")
mount_options="defaults,noatime,compress=zstd"
;;
*)
mount_options="defaults"
;;
esac
echo -e "${YELLOW}Mounting $device at $mountpoint with options: $mount_options${NC}"
if mount -o "$mount_options" "$device" "$mountpoint"; then
log_message "Mounted $device at $mountpoint"
echo -e "${GREEN}Filesystem mounted successfully${NC}"
# Add to fstab for permanent mounting
uuid=$(blkid -o value -s UUID "$device")
if [ -n "$uuid" ]; then
echo "UUID=$uuid $mountpoint $fstype $mount_options 0 2" >> /etc/fstab
echo "Added to /etc/fstab for automatic mounting"
fi
else
echo -e "${RED}Failed to mount filesystem${NC}"
return 1
fi
}
# Check filesystem health
check_filesystem_health() {
local device="$1"
if [ -z "$device" ]; then
echo "Usage: check_filesystem_health <device>"
return 1
fi
echo -e "${BLUE}=== Checking filesystem health for $device ===${NC}"
# Get filesystem type
fstype=$(blkid -o value -s TYPE "$device")
case "$fstype" in
"ext4"|"ext3"|"ext2")
echo "Running fsck for ext filesystem..."
fsck.ext4 -n "$device"
;;
"xfs")
echo "Running xfs_check..."
xfs_check "$device"
;;
"btrfs")
echo "Running btrfs check..."
btrfs check "$device"
;;
*)
echo "Running generic fsck..."
fsck -n "$device"
;;
esac
log_message "Filesystem health check completed for $device"
}
# Optimize filesystem performance
optimize_filesystem() {
local mountpoint="$1"
if [ -z "$mountpoint" ]; then
echo "Usage: optimize_filesystem <mountpoint>"
return 1
fi
echo -e "${BLUE}=== Optimizing filesystem at $mountpoint ===${NC}"
# Get device and filesystem type
device=$(df "$mountpoint" | tail -1 | awk '{print $1}')
fstype=$(df -T "$mountpoint" | tail -1 | awk '{print $2}')
case "$fstype" in
"ext4")
echo "Optimizing ext4 filesystem..."
# Defragment if needed
e4defrag -c "$mountpoint"
# Optimize inode allocation
tune2fs -o journal_data_writeback "$device"
# Enable TRIM for SSDs
if [[ $(cat /sys/block/$(basename $device | sed 's/[0-9]*$//')/queue/rotational) == "0" ]]; then
tune2fs -o discard "$device"
echo "SSD optimizations applied"
fi
;;
"xfs")
echo "Optimizing XFS filesystem..."
# XFS is generally well-optimized by default
# Check if online defragmentation is needed
xfs_fsr -v "$mountpoint"
;;
"btrfs")
echo "Optimizing Btrfs filesystem..."
# Balance filesystem
btrfs balance start "$mountpoint"
# Defragment
btrfs filesystem defragment -r "$mountpoint"
;;
esac
log_message "Filesystem optimization completed for $mountpoint"
echo -e "${GREEN}Optimization completed${NC}"
}
# Backup filesystem
backup_filesystem() {
local source="$1"
local backup_name="$2"
if [ -z "$source" ]; then
echo "Usage: backup_filesystem <source_mountpoint> [backup_name]"
return 1
fi
if [ -z "$backup_name" ]; then
backup_name="filesystem_backup_$(date +%Y%m%d_%H%M%S)"
fi
backup_path="$BACKUP_DIR/$backup_name"
echo -e "${YELLOW}Creating backup of $source to $backup_path${NC}"
# Create backup directory
mkdir -p "$backup_path"
# Use rsync for efficient backup
if rsync -avH --progress "$source/" "$backup_path/"; then
log_message "Backup created: $source -> $backup_path"
echo -e "${GREEN}Backup completed successfully${NC}"
# Create metadata file
cat > "$backup_path/.backup_info" << EOF
Source: $source
Backup Date: $(date)
Backup Size: $(du -sh "$backup_path" | cut -f1)
EOF
else
echo -e "${RED}Backup failed${NC}"
return 1
fi
}
# Monitor filesystem usage
monitor_filesystem() {
echo -e "${BLUE}=== Filesystem Usage Monitor ===${NC}"
while true; do
clear
echo "Filesystem Usage (updated every 5 seconds) - Press Ctrl+C to exit"
echo "================================================================"
df -h | grep -E "^Filesystem|^/dev"
echo ""
echo "I/O Statistics:"
iostat -x 1 1 | tail -n +4
echo ""
echo "Top 5 largest directories in /:"
du -h --max-depth=1 / 2>/dev/null | sort -hr | head -5
sleep 5
done
}
# Main menu
case "${1:-help}" in
"info")
show_filesystem_info
;;
"create")
check_root
create_filesystem "$2" "$3" "$4"
;;
"mount")
check_root
mount_filesystem "$2" "$3" "$4"
;;
"check")
check_root
check_filesystem_health "$2"
;;
"optimize")
check_root
optimize_filesystem "$2"
;;
"backup")
check_root
backup_filesystem "$2" "$3"
;;
"monitor")
monitor_filesystem
;;
"help"|*)
echo "Filesystem Manager for Ubuntu 22.04"
echo "Usage: $0 [command] [options]"
echo ""
echo "Commands:"
echo " info - Show filesystem information"
echo " create <device> <type> [label] - Create filesystem"
echo " mount <device> <mountpoint> - Mount filesystem with optimal options"
echo " check <device> - Check filesystem health"
echo " optimize <mountpoint> - Optimize filesystem performance"
echo " backup <source> [name] - Backup filesystem"
echo " monitor - Monitor filesystem usage"
echo ""
echo "Supported filesystem types: ext4, xfs, btrfs"
echo ""
echo "Examples:"
echo " $0 create /dev/sdb1 ext4 MyData"
echo " $0 mount /dev/sdb1 /mnt/data"
echo " $0 check /dev/sdb1"
echo " $0 optimize /home"
echo " $0 backup /home home_backup"
;;
esac
Best Practices for File Systems
Choosing the Right Filesystem
Decision Matrix:
Use Case | Recommended Filesystem
----------------------------|----------------------
General desktop/server | ext4
Large files/databases | XFS
Snapshots/modern features | Btrfs
Maximum data integrity | ZFS
Cross-platform compatibility| FAT32/NTFS
High-performance computing | XFS or ext4
Performance Optimization
Mount Options:
# For general use mount -o defaults,noatime /dev/sdb1 /mnt/data # For SSDs mount -o defaults,noatime,discard /dev/sdb1 /mnt/data # For databases mount -o defaults,noatime,barrier=0 /dev/sdb1 /mnt/database
I/O Scheduler:
# For SSDs echo none > /sys/block/sda/queue/scheduler # For HDDs echo mq-deadline > /sys/block/sda/queue/scheduler
Filesystem Tuning:
# Optimize ext4 for SSDs tune2fs -o discard /dev/sda1 # Reduce reserved space (ext4) tune2fs -m 1 /dev/sda1
Security and Maintenance
Regular Checks:
# Schedule regular filesystem checks echo "0 3 * * 0 /sbin/fsck -A -f" >> /etc/crontab
Backup Strategy:
# Automated backups rsync -avH /home/ /backup/home/ # Filesystem snapshots (Btrfs) btrfs subvolume snapshot /home /home/.snapshots/$(date +%Y%m%d)
Monitoring:
# Set up disk usage alerts echo "df -h | awk 'NR>1 {if(\$5+0 > 85) print \$0}' | mail -s 'Disk Alert' admin@example.com" | crontab -